Displaying content based on viewing direction

ABSTRACT

A computer detects a person in an environment. The computer determines a view direction of the person. The computer displays content relevant to the determined view direction of the person.

TECHNICAL FIELD

The present invention relates to displaying content, and moreparticularly to displaying content based on viewing direction.

BACKGROUND

In the digital age, people desire for information to be available attheir fingertips at all times. The invention of the Internet has made itpossible for a person to, within seconds, utilize a search engine tofind and learn almost anything. This has been furthered with the arrivalof the smartphone, which has allowed user to gain access to almost anyinformation no matter their location. Smart watches, glasses, and otherproducts have helped further satisfy this thirst for real-time access toinformation.

SUMMARY

The present invention provides a method, system, and computer programproduct for determining a view direction of a user. A computer detects aperson in an environment. The computer determines a view direction ofthe person. The computer displays content relevant to the determinedview direction of the person.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a display system, in accordance with an embodiment ofthe invention.

FIG. 2 is a flowchart illustrating the operations of the content displayprogram of FIG. 1, in accordance with an embodiment of the invention.

FIG. 3 is an illustration of the determination of a view direction, inaccordance with an embodiment of the invention.

FIG. 4 is an illustration of relevant content being displayed on atransparent display, in accordance with an embodiment of the invention.

FIG. 5 is a block diagram depicting the hardware components of thedisplay system of FIG. 1, in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described in detailwith reference to the accompanying Figures.

FIG. 1 illustrates display system 100, in accordance with an embodimentof the invention. In an exemplary embodiment, display system 100includes computing device 110, camera 120, and camera 130 allinterconnected via network 140.

Camera 120 is a computing device that is capable of taking videos and/orphotographs of an environment. In the example embodiment, camera 120 iscapable of zooming in and zooming out in order to fully capture anobject in an environment. In addition, in the example embodiment, camera120 is capable of detecting an object, such as a person, entering anenvironment by way of utilizing one or more motion detectors.

Camera 130 is a computing device that is capable of taking videos and/orphotographs of an environment. In the example embodiment, camera 130 iscapable of zooming in and zooming out in order to fully capture anobject in an environment. In addition, in the example embodiment, camera130 is capable of detecting an object, such as a person, entering anenvironment by way of utilizing one or more motion detectors.

Computing device 110 includes content display program 112 and userinterface 114. Computing device 110 may be a desktop computer, anotebook, a laptop computer, a tablet computer, a handheld device, asmart-phone, a thin client, or any other electronic device or computingsystem capable of receiving and sending data to and from other computingdevices, such as camera 120 and camera 130, via a network, such asnetwork 140. Although not shown, optionally, computing device 110 cancomprise a cluster of web devices executing the same software tocollectively process requests. In the example embodiment, computingdevice 110 also includes a transparent display, such as a window, whichone or more users may interact with using, for example, a gesture orhand motion. Furthermore, in other embodiments, computing device 110 mayinclude one or more hard wired cameras rather than being connected toone or more cameras via a network. Computing device 110 is described inmore detail with reference to FIG. 3.

User interface 114 includes components used to receive input from a userand transmit the input to an application residing on computing device110. In the example embodiment, user interface 114 uses a combination oftechnologies and devices, such as device drivers, to provide a platformto enable users of computing device 110 to interact with content displayprogram 112. In the example embodiment, user interface 114 receivesinput, such as input received via a gesture, hand motion or interactionwith a transparent display (which functions similar to a touch screen).

Content display program 112 is a program capable of utilizing camera 120and camera 130 in order to capture images of one or more people in anenvironment. In the example embodiment, content display program 112 iscapable of determining a view direction for each of the one or morepeople in the environment. Furthermore, in the example embodiment,content display program 112 is capable of displaying content on adisplay, such as a transparent display, based on the view direction. Theoperations and functions of content display program 112 are described inmore detail with reference to FIG. 2.

FIG. 2 is a flowchart illustrating the operations of content displayprogram 112 in determining a view direction of one or more people in anenvironment and displaying content based on the view direction, inaccordance with an embodiment of the invention. In the exampleembodiment, content display program 112 detects a person in anenvironment (step 202). In the example embodiment, content displayprogram 112 utilizes camera 120 and/or camera 130 to detect a person (ormore than one person) in the environment. In further embodiments,content display program 112 may utilize motion detecting sensors,infrared sensors or other detecting mechanisms to detect a person in theenvironment.

Content display program 112 determines the view direction of the persondetected in the environment (step 204). In the example embodiment,content display program 112 determines a view direction of the person(s)by utilizing geometric principles in order to determine an object or anarea that the person is viewing. In the example embodiment, wherecomputing device 110 includes a transparent display, such as a window,if the person is looking out the window at an object or an area, contentdisplay program 112 may utilize geometric principles to determine anobject or an area that the person is viewing while accounting for thethickness of the glass/material of window by factoring in a lightrefraction analysis. For example, content display program 112 mayutilize a camera, such as camera 120, that is closest in proximity tothe person, to determine the direction of the gaze of the person withrespect to the transparent display. Content display program 112 thentakes into account the refractive index of the material to adjust theviewing angle (with respect to the transparent display). Additionally,another camera(s) may be utilized in order to validate the determineddirection of the gaze of the person. This analysis will be described infurther detail below with regard to FIG. 3.

In further embodiments, content display program 112 may utilize eye/gazetracking technology in order to determine the view direction of theperson. For example, if a special lens or film is affixed to the corneaof the person, content display program 112 may communicate with thespecial lens via network 140 in order to determine the view direction.The special lens or film may include precise position sensors to followphysical movements of the eye. In addition, a tiny mirror orelectromechanical transducer may be embedded in the special lens orfilm, with the mirror/transducer using light beams or electromagneticfields to quantify the eye's orientation and follow changes in the gazeposition.

In even further embodiments, eye position and movement may be detectedwithout the use of attachments to the cornea. For example, contentdisplay program 112 may utilize a microprojector to transmit an infraredbeam at the eye, with the reflection patterns being picked up by a setof sensors. In this embodiment, the reflections may occur from thecornea or from the retina as the infrared beam passes through the lens,into the eye, and back out.

Content display program 112 displays content based on the determinedview direction (step 206). In the example embodiment, where computingdevice 110 includes a transparent display, such as a window, contentdisplay program 112 may display the content in an area on thetransparent display so that the person(s) may view the content whilelooking through the transparent display. In addition, in embodimentswhere more than one person is detected in the environment, contentdisplay program 112 may display content in areas of the transparentdisplay relevant to each person. For example, if the view direction of afirst person is focused on a first building, while the view direction ofthe second person is focused on a downtown hotel, content displayprogram 112 may display content relevant to the first building on anarea of the transparent display that is within close proximity to thefirst person, while simultaneously displaying content relevant to thedowntown hotel on an area of the transparent display that is withinclose proximity to the second person. Furthermore, in the exampleembodiment, a user/person may manipulate the displayed content by way ofa gesture, hand motion, utilization of a keyboard/touchscreen, and/orother type of user input. For example, a user/person may use an upwardhand motion to scroll down through the content. In addition, in theexample embodiment, content display program 112 may display the relevantcontent for a specific period of time, as set by an administrator.

Content display program 112 determines whether the view direction of theperson(s) detected in the environment has changed (decision 208). In theexample embodiment, content display program 112 determines whether theview direction of the person(s) detected in the environment has changedby utilizing cameras (such as camera 120 and camera 130) to track theperson(s) gaze in the same manner as described above. If content displayprogram 112 determines that the view direction of the person(s) detectedin the environment has not changed (decision 208, “NO” branch), contentdisplay program 112 does not take any additional steps and continuesdisplaying the relevant content.

If content display program 112 determines that the view direction of theperson(s) detected in the environment has changed (decision 208, “YES”branch), content display program 112 determines the updated viewdirection of the person(s) in the same manner as described above andupdates the display (transparent display) so that content relevant tothe focus of the updated view direction is displayed (step 210).

FIG. 3 is an illustration of the determination of a view direction of anindividual, in accordance with an embodiment of the invention. In theexample embodiment, FIG. 3 depicts a focus of an individual based onview direction information determined by utilizing camera 120 and/orcamera 130. Extrapolation 302 represents a simple extrapolation of thefocus of an individual without taking refraction into account (i.e.,refractive index of material that transparent display 301 is composedof). Angle of incidence 306(θi) represents the angle of incidence of theview direction while, angle of refraction 308(θ_(R)) represents theangle of refraction, taking into account the material that transparentdisplay 301 is composed of. In the example embodiment, the angle ofincidence may be determined by utilizing geometric techniques frominformation retrieved by utilizing camera 120 and/or camera 130. Inorder to determine the angle of refraction, the equation below may beutilized:n _(i) sin(θ_(i))=n _(r) sin(θ_(r))  (1)

Therefore, for an environment where a person(s) is looking through atransparent display n_(i) represents the index of refraction of theincident medium (for example, air), and n_(r) represents the index ofrefraction of the refractive medium (for example, glass). Once the angleof incidence is determined as described above, the angle of refractionmay be determined by utilizing equation 1.

Content display program 112 may then determine a point of focus throughthe transparent display 301, by way of factoring in the angle ofrefraction, and also factoring in the thickness of transparent display301.

FIG. 4 is an illustration of relevant content being displayed on atransparent display, in accordance with an embodiment of the invention.In the example embodiment, both users 408 and 410 are focused on thesame object 412. Therefore, after determining the focus of each user,content display program 112 displays information relevant to object 412on transparent display 402, as displayed by information 404 andinformation 406. In the example embodiment, as stated above, contentdisplay program 112 determines the focus (view direction) of a user andthen displays information relevant to the user, however, content displayprogram 112 may also determine the focus of all users in the environmentprior to displaying relevant information on transparent display 402.

The foregoing description of various embodiments of the presentinvention has been presented for purposes of illustration anddescription. It is not intended to be exhaustive nor to limit theinvention to the precise form disclosed. Many modifications andvariations are possible. Such modifications and variations that may beapparent to a person skilled in the art of the invention are intended tobe included within the scope of the invention as defined by theaccompanying claims.

FIG. 5 depicts a block diagram of components of computing device 110,camera 120, and camera 130 of display system 100 in accordance with anillustrative embodiment of the present invention. It should beappreciated that FIG. 5 provides only an illustration of oneimplementation and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environment may be made.

Computing device 110, camera 120, and camera 130 include communicationsfabric 502, which provides communications between computer processor(s)504, memory 506, persistent storage 508, communications unit 512, andinput/output (I/O) interface(s) 514. Communications fabric 502 can beimplemented with any architecture designed for passing data and/orcontrol information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware components within a system. For example,communications fabric 502 can be implemented with one or more buses.

Memory 506 and persistent storage 508 are computer-readable storagemedia. In this embodiment, memory 506 includes random access memory(RAM) 516 and cache memory 518. In general, memory 506 can include anysuitable volatile or non-volatile computer-readable storage media.

Content display program 112 and user interface 114 in computing device110 may be stored in persistent storage 508 for execution by one or moreof the respective computer processors 504 via one or more memories ofmemory 506. In this embodiment, persistent storage 508 includes amagnetic hard disk drive. Alternatively, or in addition to a magnetichard disk drive, persistent storage 508 can include a solid state harddrive, a semiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer-readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 508 may also be removable. Forexample, a removable hard drive may be used for persistent storage 508.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer-readable storage medium that is also part of persistent storage508.

Communications unit 512, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 512 includes one or more network interface cards.Communications unit 512 may provide communications through the use ofeither or both physical and wireless communications links. Contentdisplay program 112 and user interface 114 in computing device 110 maybe downloaded to persistent storage 508 through communications unit 512.

I/O interface(s) 514 allows for input and output of data with otherdevices that may be connected to computing device 110, camera 120, andcamera 130. For example, I/O interface 514 may provide a connection toexternal devices 520 such as, a keyboard, keypad, a touch screen, and/orsome other suitable input device. External devices 520 can also includeportable computer-readable storage media such as, for example, thumbdrives, portable optical or magnetic disks, and memory cards. Softwareand data used to practice embodiments of the present invention, e.g.,content display program 112 and user interface 114 in computing device110, can be stored on such portable computer-readable storage media andcan be loaded onto persistent storage 508 via I/O interface(s) 514. I/Ointerface(s) 514 can also connect to a display 522.

Display 522 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature. The presentinvention may be a system, a method, and/or a computer program product.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge devices. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or device. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A computer program product for determining a viewdirection of a user, the computer program product comprising: one ormore computer-readable storage media and program instructions stored onthe one or more computer-readable storage media, the programinstructions comprising: program instructions to detect a person in anenvironment by utilizing motion detecting sensors and infrared sensors;based on detecting a person in the environment: program instructions todetermine an angle of incidence, wherein determining an angle ofincidence further comprises utilizing a microprojector which transmitsan infrared beam to an eye of the person, and utilizing a set of sensorsto pick up the reflection patterns; program instructions to determine anangle of refraction based on an index of refraction of a transparentdisplay; program instructions to determine a view direction of theperson based on the determined angle of incidence, the determined angleof refraction, and a thickness of the transparent display; and programinstructions to display content relevant to the determined viewdirection of the person.